{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "numeric-luxury",
   "metadata": {},
   "source": [
    "# Explainable Boosting Machine\n",
    "\n",
    "Links to API References: [ExplainableBoostingClassifier](./python/api/ExplainableBoostingClassifier.ipynb), [ExplainableBoostingRegressor](./python/api/ExplainableBoostingRegressor.ipynb)\n",
    "\n",
    "*See the reference paper for full details [[1](lou2013accurate_ebm)].*  [Link](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd13.pdf)\n",
    "\n",
    "<h2>Summary</h2>\n",
    "\n",
    "Explainable Boosting Machine (EBM) is a tree-based, cyclic gradient boosting Generalized Additive Model with automatic interaction detection. EBMs are often as accurate as state-of-the-art blackbox models while remaining completely interpretable.\n",
    "\n",
    "<h2>How it Works</h2>\n",
    "\n",
    "As part of the framework, InterpretML also includes a new interpretability algorithm -- the Explainable Boosting Machine (EBM). EBM is a glassbox model, designed to have accuracy comparable to state-of-the-art machine learning methods like Random Forest and Boosted Trees, while being highly intelligibile and explainable. EBM is a generalized additive model (GAM) of the form:\n",
    "\n",
    "$$\n",
    "g(E[y]) = \\beta_0 + \\sum f_j(x_j)\n",
    "$$\n",
    "\n",
    "where $g$ is the link function that adapts the GAM to different settings such as regression or classification.\n",
    "\n",
    "EBM has a few major improvements over traditional GAMs [[2](hastie1987generalized_ebm)]. First, EBM learns each feature function $f_j$ using modern machine learning techniques such as bagging and gradient boosting. The boosting procedure is carefully restricted to train on one feature at a time in round-robin fashion using a very low learning rate so that feature order does not matter. It round-robin cycles through features to mitigate the effects of co-linearity and to learn the best feature function $f_j$ for each feature to show how each feature contributes to the model's prediction for the problem. Second, EBM can automatically detect and include pairwise interaction terms of the form:\n",
    "\n",
    "$$\n",
    "g(E[y]) = \\beta_0 + \\sum f_i(x_i)  + \\sum f_{i,j}(x_i,x_j) \n",
    "$$\n",
    "\n",
    "which further increases accuracy while maintaining intelligibility. EBM is a fast implementation of the GA2M algorithm [[1](lou2013accurate_ebm)], written in C++ and Python. The implementation is parallelizable, and takes advantage of joblib to provide multi-core and multi-machine parallelization. The algorithmic details for the training procedure, selection of pairwise interaction terms, and case studies can be found in [[1](lou2013accurate_ebm), [3](lou2012intelligible_ebm), [4](caruana2015intelligible_ebm)].\n",
    "\n",
    "EBMs are highly intelligible because the contribution of each feature to a final prediction can be visualized and understood by plotting $f_j$. Because EBM is an additive model, each feature contributes to predictions in a modular way that makes it easy to reason about the contribution of each feature to the prediction.\n",
    "\n",
    "To make individual predictions, each function $f_j$ acts as a lookup table per feature, and returns a term contribution. These term contributions are simply added up, and passed through the link function $g$ to compute the final prediction. Because of the modularity (additivity), term contributions can be sorted and visualized to show which features had the most impact on any individual prediction.\n",
    "\n",
    "To keep the individual terms additive, EBM pays an additional training cost, making it somewhat slower than similar methods. However, because making predictions involves simple additions and lookups inside of the feature functions $f_j$, EBMs are one of the fastest models to execute at prediction time. EBM's light memory usage and fast predict times makes it particularly attractive for model deployment in production.\n",
    "\n",
    "If you find video as a better medium for learning the algorithm, you can find a conceptual overview of the algorithm below:\n",
    "[![Video for explaining how EBM works.](https://img.youtube.com/vi/MREiHgHgl0k/0.jpg)](https://www.youtube.com/watch?v=MREiHgHgl0k \"The Science Behind InterpretML: Explainable Boosting Machine\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "announced-warning",
   "metadata": {},
   "source": [
    "<h2>Code Example</h2>\n",
    "\n",
    "The following code will train an EBM classifier for the adult income dataset. The visualizations provided will be for both global and local explanations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bibliographic-mouse",
   "metadata": {},
   "outputs": [],
   "source": [
    "from interpret import set_visualize_provider\n",
    "from interpret.provider import InlineProvider\n",
    "set_visualize_provider(InlineProvider())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "packed-excess",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "from interpret.glassbox import ExplainableBoostingClassifier\n",
    "from interpret import show\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\",\n",
    "    header=None)\n",
    "df.columns = [\n",
    "    \"Age\", \"WorkClass\", \"fnlwgt\", \"Education\", \"EducationNum\",\n",
    "    \"MaritalStatus\", \"Occupation\", \"Relationship\", \"Race\", \"Gender\",\n",
    "    \"CapitalGain\", \"CapitalLoss\", \"HoursPerWeek\", \"NativeCountry\", \"Income\"\n",
    "]\n",
    "X = df.iloc[:, :-1]\n",
    "y = df.iloc[:, -1]\n",
    "\n",
    "seed = 42\n",
    "np.random.seed(seed)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)\n",
    "\n",
    "ebm = ExplainableBoostingClassifier()\n",
    "ebm.fit(X_train, y_train)\n",
    "\n",
    "auc = roc_auc_score(y_test, ebm.predict_proba(X_test)[:, 1])\n",
    "print(\"AUC: {:.3f}\".format(auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3819596",
   "metadata": {},
   "outputs": [],
   "source": [
    "show(ebm.explain_global())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bsdfasdfdsb",
   "metadata": {},
   "source": [
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "kgkwkjweks",
   "metadata": {},
   "outputs": [],
   "source": [
    "show(ebm.explain_local(X_test[:5], y_test[:5]), 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "asdfasdfdsa",
   "metadata": {},
   "source": [
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "occupied-withdrawal",
   "metadata": {},
   "source": [
    "<h2>Further Resources</h2>\n",
    "\n",
    "- [Paper: GA2M](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd12.pdf)\n",
    "- [Paper: InterpretML Framework](https://arxiv.org/pdf/1909.09223.pdf)\n",
    "- [Video: The Science Behind InterpretML: Explainable Boosting Machine](https://www.youtube.com/watch?v=MREiHgHgl0k)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaging-string",
   "metadata": {},
   "source": [
    "<h2>Bibliography</h2>\n",
    "\n",
    "(lou2013accurate_ebm)=\n",
    "[1] Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, 623–631. 2013. [Paper Link](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd13.pdf)\n",
    "\n",
    "(hastie1987generalized_ebm)=\n",
    "[2] Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371–386, 1987.\n",
    "\n",
    "(lou2012intelligible_ebm)=\n",
    "[3] Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 150–158. 2012. [Paper Link](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd12.pdf)\n",
    "\n",
    "(caruana2015intelligible_ebm)=\n",
    "[4] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, 1721–1730. 2015. [Paper Link](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/KDD2015FinalDraftIntelligibleModels4HealthCare_igt143e-caruanaA.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ruled-shell",
   "metadata": {},
   "source": [
    "[^params]: All models were trained with their default parameters. EBM's current default parameters are chosen for computational speed, to enable ease of experimentation. For the best accuracy and interpretability, we recommend using reference parameters: 100 inner bags, 100 outer bags, 5000 epochs, and a learning rate of 0.01"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
